System and method for displaying objects of interest at an incident scene

ABSTRACT

A system and method for displaying an image of an object of interest located at an incident scene. The method includes receiving, from the image capture device, a first video stream of the incident scene, and displaying the video stream. The method includes receiving an input indicating a pixel location in the video stream, and detecting the object of interest in the video stream based on the pixel location. The method includes determining an object class, an object identifier, and metadata for the object of interest. The metadata includes the object class, an object location, an incident identifier corresponding to the incident scene, and a time stamp. The method includes receiving an annotation input for the object of interest, and associating the annotation input and the metadata with the object identifier. The method includes storing, in a memory, the object of interest, the annotation input, and the metadata.

BACKGROUND OF THE INVENTION

Public safety personnel responding to an incident scene (for example, acrime scene, a traffic accident, and the like) may document the incidentscene for later use as evidence in criminal trials or other officialinvestigations or proceedings. Personnel document the scene itself (forexample, an intersection, an outdoor area, a space within a building,and the like) as well as objects of interest located at the incidentscene. An object of interest is any object present at the incident sceneand related to the incident (for example, involved in or relevant to aninvestigation of the incident). Objects of interest may include, forexample, automobiles (for example, in the case of a traffic accident)and weapons (for example, in the case of a crime scene). Public safetypersonnel may document the incident scene and objects of interest byusing portable electronic devices (for example, body-worn cameras, smarttelephones, portable radios, and the like) to record still images andvideo of the object of interest, or to record audio or text describingthe object of interest.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS

The accompanying figures, where like reference numerals refer toidentical or functionally similar elements throughout the separateviews, together with the detailed description below, are incorporated inand form part of the specification, and serve to further illustrateembodiments of concepts that include the claimed invention, and explainvarious principles and advantages of those embodiments.

FIG. 1 is a diagram of system for a displaying an image of an object ofinterest located at an incident scene in accordance with someembodiments.

FIG. 2 is a diagram of a mobile electronic device in accordance withsome embodiments.

FIG. 3 is a flowchart of a method for selecting and annotating an objectof interest located at an incident scene in accordance with someembodiments.

FIG. 4 is a graphical user interface in accordance with someembodiments.

FIG. 5 is a flowchart of a method for displaying an image of an objectof interest located at an incident scene in accordance with someembodiments.

FIG. 6A is a graphical user interface in accordance with someembodiments.

FIG. 6B is a graphical user interface in accordance with someembodiments.

FIG. 7 is a flowchart of a method for highlighting a visual change in animage of an object of interest located at an incident scene inaccordance with some embodiments.

FIG. 8A is a graphical user interface in accordance with someembodiments.

FIG. 8B is a graphical user interface in accordance with someembodiments.

FIG. 9 is a flowchart of a method for annotating an incident scene inaccordance with some embodiments.

FIG. 10 is a graphical user interface in accordance with someembodiments.

Skilled artisans will appreciate that elements in the figures areillustrated for simplicity and clarity and have not necessarily beendrawn to scale. For example, the dimensions of some of the elements inthe figures may be exaggerated relative to other elements to help toimprove understanding of embodiments of the present invention.

The apparatus and method components have been represented whereappropriate by conventional symbols in the drawings, showing only thosespecific details that are pertinent to understanding the embodiments ofthe present invention so as not to obscure the disclosure with detailsthat will be readily apparent to those of ordinary skill in the arthaving the benefit of the description herein.

DETAILED DESCRIPTION OF THE INVENTION

Public safety personnel (for example, first responders, investigators,and the liked) responding to an incident scene may document the incidentscene and the objects of interest located at the incident scene. Publicsafety personnel may document the incident scene using portableelectronic devices to record still images or video of the object ofinterest, to record audio or text describing the object of interest, orsome combination of the foregoing. In some cases, objects of interestare removed from the incident scene to, for example, clean up the sceneor to be used later use as evidence in criminal trials or other officialinvestigations or proceedings. Investigations of incident scenes may bestopped and restarted, and may be performed by multiple personnel atmany times. An investigation of an incident scene may also be restartedafter the incident scene has been fully documented and all objects ofinterest documented and removed. An investigator may wish to re-createan incident scene, or may wish to compare multiple incidents thatoccurred at the same scene. Current systems and methods for documentingobjects of interest are inefficient in such investigations. Accordingly,because current systems and methods do not provide selection,recognition, and annotation of an object of interest from a live scenebased on user input, systems and methods are provided herein for, amongother things, displaying an image of an object of interest located at anincident scene.

One example embodiment provides a system for displaying an image of anobject of interest located at an incident scene. The system includes animage capture device, a display, a memory, and an electronic processorcoupled to the image capture device, the display, and the memory. Theelectronic processor is configured to receive, from the image capturedevice, a first video stream of the incident scene. The electronicprocessor is configured to display the first video stream on thedisplay. The electronic processor is configured to receive an inputindicating a pixel location in the first video stream. The electronicprocessor is configured to detect the object of interest in the firstvideo stream based on the pixel location. The electronic processor isconfigured to determine an object class for the object of interest. Theelectronic processor is configured to determine an object identifier forthe object of interest. The electronic processor is configured todetermine metadata for the object of interest including the objectclass, an object location, an incident identifier corresponding to theincident scene, and a time stamp. The electronic processor is configuredto receive an annotation input for the object of interest. Theelectronic processor is configured to associate the annotation input andthe metadata with the object identifier. The electronic processor isconfigured to store, in the memory, the object of interest, theannotation input, and the metadata.

Another example embodiment provides a method for displaying an image ofan object of interest located at an incident scene. The method includesreceiving, from the image capture device, a first video stream of theincident scene. The method includes displaying the first video stream onthe display. The method includes receiving an input indicating a pixellocation in the first video stream. The method includes detecting theobject of interest in the first video stream based on the pixellocation. The method includes determining an object class for the objectof interest. The method includes determining an object identifier forthe object of interest. The method includes determining metadata for theobject of interest including the object class, an object location, anincident identifier corresponding to the incident scene, and a timestamp. The method includes receiving an annotation input for the objectof interest. The method includes associating the annotation input andthe metadata with the object identifier. The method includes storing, ina memory, the object of interest, the annotation input, and themetadata.

For ease of description, some or all of the example systems presentedherein are illustrated with a single exemplar of each of its componentparts. Some examples may not describe or illustrate all components ofthe systems. Other example embodiments may include more or fewer of eachof the illustrated components, may combine some components, or mayinclude additional or alternative components.

FIG. 1 is a block diagram of a system 100 for displaying an image of anobject of interest located at an incident scene. In the exampleillustrated, the system 100 includes a portable electronic device 102, aserver 104, a database 106, and a network 108. The portable electronicdevice 102 and the server 104 are communicatively coupled via thenetwork 108. The network 108 is a communications network includingwireless and wired connections. The network 108 may be implemented usinga land mobile radio (LMR) network, and a cellular network (for example,a Long Term Evolution (LTE) network). However, the concepts andtechniques embodied and described herein may be used with networks usingother protocols, for example, Global System for Mobile Communications(or Groupe Special Mobile (GSM)) networks, Code Division Multiple Access(CDMA) networks, Evolution-Data Optimized (EV-DO) networks, EnhancedData Rates for GSM Evolution (EDGE) networks, 3G networks, 4G networks,combinations or derivatives thereof, and other suitable networks,including future-developed network architectures. In some embodiments,communications with other external devices (not shown) occur over thenetwork 108.

The portable electronic device 102, described more particularly belowwith respect to FIG. 2 , is a wireless communication device thatincludes hardware and software that enable it to communicate via thenetwork 108. The portable electronic device 102 includes an imagecapture device (for example, a camera), and is capable of capturing,storing, analyzing, displaying, and transmitting captured images of theincident scene 110, including images of the object of interest 112. Theportable electronic device 102 operates using, among other things,augmented reality technology, where live images are captured by theimage capture device and displayed (for example, on a screen) with text,graphics, or graphical user interface elements superimposed on orotherwise combined with the live images. As described in detail below,the superimposed text or graphics may be used to record or conveyinformation about the incident scene 110, the object of interest 112, orboth.

The incident scene 110 is the scene of an incident to which publicsafety personnel may respond (for example, the scene of a trafficaccident or a crime scene). The incident scene 110 may be locatedindoors or outdoors. The object of interest 112 may be any objectpresent at the incident scene, which object is related to the incident(for example, involved in or relevant to an investigation of theincident). Objects of interest may include, for example, automobiles(for example, in the case of a traffic accident) and weapons (forexample, in the case of a crime scene). Objects of interest may also betangible things not commonly thought of as objects, but which are stillremovable or transitory in nature (for example, fluids leaked fromautomobiles, debris from damaged property, blood stains, broken glass,skid marks, and fingerprints). In some embodiments, a person (forexample, a crime or accident victim, persons gathered at the scene, andthe like) may also be an object of interest. The incident scene 110 mayinclude more than one object of interest 112.

The server 104 is a computer server that includes an electronicprocessor (for example, a microprocessor, or other electroniccontroller), a memory, a network interface, and other various modulescoupled directly, by one or more control or data buses, or a combinationthereof. The memory may include read-only memory, random access memory,other non-transitory computer-readable media, or a combination thereof.The electronic processor is configured to retrieve instructions and datafrom the memory and execute, among other things, instructions to performthe methods described herein. The server 104 sends and receives dataover the network 108 using the network interface.

The server 104 reads and writes data to and from the database 106. Asillustrated in FIG. 1 , the database 106 may be a database housed on asuitable database server communicatively coupled to and accessible bythe server 104. In alternative embodiments, the database 106 may be partof a cloud-based database system external to the system 100 andaccessible by the server 104 and the portable electronic device 102 overone or more additional networks. In some embodiments, all or part of thedatabase 106 may be locally stored on the server 104. In someembodiments, as described below, the database 106 electronically storesdata on objects of interest (for example, the object of interest 112)and annotations for the objects of interest. In some embodiments, theserver 104 and the database 106 are part of a computer-aided dispatchsystem.

FIG. 2 is a diagram of an example of the portable electronic device 102.In the embodiment illustrated, the portable electronic device 102includes an electronic processor 205, a memory 210, an input/outputinterface 215, a baseband processor 220, a transceiver 225, an antenna230, a microphone 235, a camera 240, and a display 245. The illustratedcomponents, along with other various modules and components are coupledto each other by or through one or more control or data buses thatenable communication therebetween. The use of control and data buses forthe interconnection between and exchange of information among thevarious modules and components would be apparent to a person skilled inthe art in view of the description provided herein.

The electronic processor 205 obtains and provides information (forexample, from the memory 210 and/or the input/output interface 215), andprocesses the information by executing one or more software instructionsor modules, capable of being stored, for example, in a random accessmemory (“RAM”) area of the memory 210 or a read only memory (“ROM”) ofthe memory 210 or another non-transitory computer readable medium (notshown). The software can include firmware, one or more applications,program data, filters, rules, one or more program modules, and otherexecutable instructions. The electronic processor 205 is configured toretrieve from the memory 210 and execute, among other things, softwarerelated to the control processes and methods described herein.

The memory 210 can include one or more non-transitory computer-readablemedia, and includes a program storage area and a data storage area. Theprogram storage area and the data storage area can include combinationsof different types of memory, as described herein. In the embodimentillustrated, the memory 210 stores, among other things, metadata 250 andannotation input 255 (both described in detail below), and an objectclassifier 260. As described in detail below, the object classifier 260(for example, a Haar feature-based cascade classifier) may be executedby the electronic processor 205 to electronically detect and classifyobjects within images and video streams captured by the camera 240.

The input/output interface 215 is configured to receive input and toprovide system output. The input/output interface 215 obtainsinformation and signals from, and provides information and signals to,(for example, over one or more wired and/or wireless connections)devices both internal and external to the portable electronic device102.

The electronic processor 205 is configured to control the basebandprocessor 220 and the transceiver 225 to transmit and receive video andother data to and from the portable electronic device 102. The basebandprocessor 220 encodes and decodes digital data sent and received by thetransceiver 225. The transceiver 225 transmits and receives radiosignals to and from various wireless communications networks (forexample, the network 108) using the antenna 230. The electronicprocessor 205, the baseband processor 220, and the transceiver 225 mayinclude various digital and analog components, which for brevity are notdescribed herein and which may be implemented in hardware, software, ora combination of both. Some embodiments include separate transmittingand receiving components, for example, a transmitter and a receiver,instead of a combined transceiver 225.

The microphone 235 is capable of sensing sound, converting the sound toelectrical signals, and transmitting the electrical signals to theelectronic processor 205 via the input/output interface 215. Theelectronic processor 205 processes the electrical signals received fromthe microphone 235 to, for example, produce an audio stream.

The camera 240 is an image capture device for capturing images and videostreams, including a portion or the entire incident scene 110, by, forexample, sensing light in at least the visible spectrum. The camera 240communicates the captured images and video streams to the electronicprocessor 205 via the input/output interface 215. It should be notedthat the terms “image” and “images,” as used herein, may refer to one ormore digital images captured by the camera 240, or processed by theelectronic processor 205, or displayed on the display 245. Further, theterms “image” and “images,” as used herein, may refer to still images orsequences of images (that is, a video stream). As illustrated, thecamera 240 is integrated into the portable electronic device 102. Inalternative embodiments, the camera 240 is separate from the portableelectronic device 102, and communicates captured images to the portableelectronic device 102 via a wired or wireless connection. For example,the camera 240 may be integrated into a body-worn camera or a vehicle'sdash or roof mount camera, which communicates with the portableelectronic device 102.

In some embodiments, the camera 240 may be a stereoscopic camera, or theportable electronic device 102 may include a stereoscopic camera. Insuch embodiments, the portable electronic device 102 can capturethree-dimensional information about the incident scene 110 and theobject of interest 112. In some embodiments, three-dimensionalinformation may be captured using radar sensors or infrared rangingsensors (not shown).

The display 245 is a suitable display such as, for example, a liquidcrystal display (LCD) touch screen, or an organic light-emitting diode(OLED) touch screen. The portable electronic device 102 implements agraphical user interface (GUI) (for example, generated by the electronicprocessor 205, from instructions and data stored in the memory 210, andpresented on the display 245), that enables a user to interact with theportable electronic device 102. In some embodiments, the portableelectronic device 102 operates or is integrated with a head-mounteddisplay (HMD) or an optical head-mounted display (OHMD). In someembodiments, the portable electronic device 102 operates or isintegrated with a LCD touch screen console display or heads up display(HUD) in a vehicle.

As described in detail below, the portable electronic device 102 iscapable of receiving and processing images captured by the camera 240,and displaying processed images in a graphical user interface on thedisplay 245. Computerized image capturing and processing techniques areknown, and will not be described in detail.

In some embodiments, the portable electronic device 102 is a smarttelephone. In other embodiments, the portable electronic device 102 maybe a tablet computer, a vehicle's dash console, a smart watch, aportable radio, or another portable or mobile electronic devicecontaining software and hardware enabling it to operate as describedherein.

Returning to FIG. 1 , an investigator responding to the incident scene110, using the portable electronic device 102 may wish to document theobject of interest 112. Accordingly, FIG. 3 illustrates an examplemethod 300 for selecting and annotating the object of interest 112 atthe incident scene 110. The method 300 is described with respect to FIG.4 , which illustrates the incident scene 110 and a graphical userinterface displayed on the portable electronic device 102. The method300 is described as being performed by the portable electronic device102 and, in particular, the electronic processor 205. However, it shouldbe understood that in some embodiments, portions of the method 300 maybe performed by other devices, including for example, the server 104.

At block 302, the electronic processor 205 receives, from the camera240, a first video stream 402 of the incident scene 110. At block 304,the electronic processor 205 controls the display of the first videostream 402 on the display 245. In some embodiments, the portableelectronic device 102 continually captures and displays video streams ofthe incident scene 110, for example, as in an augmented reality display.

At block 305, the electronic processor 205 receives an input indicatinga pixel location in the first video stream 402 (for example, in a frameof the first video stream 402). The input may be in response to a touch,tap, or press on the display 245, which indicates one or more pixels ata pixel location in the first video stream 402. In embodiments where theportable electronic device 102 operates or is integrated with ahead-mounted display (HMD) or an optical head-mounted display (OHMD),the input may be in response to a detected hand gesture, a detected eyemovement, and the like.

At block 306, the electronic processor 205 detects the object ofinterest 112 in the first video stream 402 based on the pixel location.For example, the electronic processor 205 may direct the objectclassifier 260 to detect an object within a limited area surrounding thepixel location. In some embodiments, the object classifier 260continuously detects and classifies multiple objects in the first videostream 402, and the input and corresponding pixel location are used toselect one of the detected objects. Accordingly, it should be noted thatit is not a requirement to display the video stream 402 in order todetect an object or objects of interest. In some embodiments, multipleobjects may possibly be selected due to object recognition ambiguity,partial overlap of objects in the scene, and close proximity of objectsto each other. In these embodiments, the best match will be selected. Itshould be understood that multiple object selections may be retained forlater selection by a user. In some embodiments, the electronic processor205 determines a boundary 404 for the object of interest 112 based onthe pixel location (for example, using edge analytics).

At block 308, the electronic processor 205 determines an object classfor the object of interest. For example, the electronic processor 205may determine an object class using the object classifier 260. In theexample illustrated, the object of interest 112 is a vehicle, which hasbeen involved in a traffic accident. In this example, the object classis “vehicle.” In some embodiments, the object class may be more or lessspecific (for example, “compact car,” or “transportation”). At block310, the electronic processor 205 determines an object identifier forthe object of interest 112. The object identifier is an electronicidentifier, for example, a serial number, which may be used to uniquelyidentify the object of interest 112 in the database 106.

At block 312, the electronic processor 205 determines metadata 250 forthe object of interest including the object class, an object location,an incident identifier corresponding to the incident scene, and a timestamp. In some embodiments, the metadata 250 includes a user identifierfor the user who selected the object of interest 112. The metadata mayalso include data based on the object class. In one example, metadata250 for the object class “vehicle” may include the color, type (forexample, sedan, compact car, truck, or sport utility vehicle), and thelicense plate of the vehicle (for example, as determined by an opticalcharacter recognition analysis of the first video stream 402). Theincident identifier is a unique electronic identifier for the incidentprovided by, for example, a computer aided dispatch system. The timestamp may be, for example, the time and date when the first video stream402 is captured. The object location is the location of the object ofinterest 112 within the incident scene 110. The object location may bebased on the location of the portable electronic device 102 (forexample, as reported by a global positioning system receiver) and thelocation of the object of interest 112 relative to the portableelectronic device 102. In some embodiments, the relative location may bedetermined using image analysis, for example, by comparing portions ofthe incident scene near the object of interest 112 to items within theincident scene of a known size to determine a distance. In someembodiments, the relative location may be determined using a rangeimaging technique (for example, stereo triangulation). In someembodiments, the portable electronic device 102 may be equipped with arange-sensing device, such as, for example, an infrared transceiver fordetermining the relative location of the object of interest 112. In someembodiments, the location of the object of interest 112 within theincident scene 110 may be determined based on the distance (for example,in pixels) of particular points along the boundary 404 relative toparticular points in the first video stream 402. In some embodiments,the metadata 250 includes information indicating the orientation of theobject of interest 112 within the incident scene 110, relative to, forexample, a fixed vector in the incident scene 110, a compass direction,or the vector representing the orientation of the portable electronicdevice 102.

At block 314, the electronic processor 205 receives an annotation input255 for the object of interest. The annotation input 255 may be receivedfrom a user of the portable electronic device 102 and may include, forexample, text-based annotation, audio annotation, video annotation, andimage annotation. At block 316, the electronic processor 205 associatesthe annotation input 255 and the metadata 250 with the objectidentifier, and, at block 318, stores (for example, in the memory 210)the object of interest 112 (for example, an image or video of, areference to, or a description of the object of interest 112, or somecombination of the foregoing), the annotation input 255, and themetadata 250. In some embodiments, the object of interest 112 (that is,an image of the object of interest 112), the annotation input 255, andthe metadata 250 are communicated to the server 104 and stored in thedatabase 106 in addition to, or in place of, being stored in the memory210. As illustrated, the metadata 250 and the annotation input 255 aredisplayed by the graphical user interface of the portable electronicdevice 102 associated with the object of interest 112. In someembodiments, the annotation input 255 and the metadata 250 are notdisplayed directly. For example, an icon or icons representing theannotation input 255 and the metadata 250 may be displayed. Inputsreceived by the electronic processor 205 selecting the icons would allowa user of the portable electronic device 102 to access the annotationinput 255 and the metadata 250.

In some embodiments, the electronic processor may receive a second input(for example, a tap or touch) selecting the object of interest, and inresponse to the second input, display an executable menu based on atleast one of the incident identifier, the object identifier, andmetadata 250. For example, a second investigator may arrive on theincident scene 110 after the object of interest 112 has been annotatedand stored. The second investigator, using another mobile electronicdevice, may view the incident scene 110 and touch on the object ofinterest 112 on a display may pop up an executable menu allowing thesecond investigator to view and edit the metadata 250 or the annotationinput 255 (for example, by interacting with a touch screen or a keypad,or by providing voice-to-text inputs). In such embodiments, the metadata250 may be updated to indicate who edited the metadata 250, and when theedits were made. In some embodiments, all versions of the metadata 250are stored in order to create an audit trail. The executable menu maytrigger the display or more detailed annotations, for example, drillingdown from a summary view (for example, “the vehicle contained 4passengers”) into individual data points (for example, the names andvital statistics for the passengers).

As noted above, the object of interest 112, the annotation input 255,and the metadata 250 may be stored in the database 106. Accordingly,they may be made available for other users to access, using one or morecomputers or portable electronic devices. In such embodiments, eachdevice synchronizes with the database 106, allowing each device to haveaccess to the latest information regarding the incident scene 110 andthe object of interest 112. In addition, the annotation input 255 andthe metadata 250 may be available for viewing outside of an augmentedreality display, for example, in a list format using a note-taking orother application that may or may not be tied to the object of interest112.

Embodiments of the system 100 include more than one portable electronicdevice 102. In such embodiments, the other portable electronic devicesare able to see the annotations added according to the method 300. Forexample, the electronic processor 205 receives, from the image capturedevice, a video stream of the incident scene that includes the object ofinterest 112 (similar to the first video stream 402 of FIG. 4 ). Theelectronic processor 205 displays the video stream on the display 245,and, as described above, detects the object of interest in the videostream. When the object of interest 112 is detected, the electronicprocessor 205 retrieves, from the memory 210, the annotation input 255and the metadata 250 based on, for example, the incident identifier andthe object identifier, and displays the annotation input 255 and themetadata 250 for the object of interest.

Returning to FIG. 1 , the investigator may return to re-investigate theincident scene 110 after the initial investigation is complete and thereal-word object of interest 112 has been removed (for example, thevehicle has been towed). Accordingly, FIG. 5 illustrates an examplemethod 500 for displaying an image of the object of interest 112 at theincident scene 110. The method 500 is described with respect to FIGS. 6Aand 6B, which illustrate the incident scene 110 and a graphical userinterface displayed on the portable electronic device 102. The method500 is described as being performed by the portable electronic device102 and, in particular, the electronic processor 205. However, it shouldbe understood that in some embodiments, portions of the method 500 maybe performed by other devices, including for example, the server 104.

At block 502, the electronic processor 205 receives, from the camera240, a second video stream 602 of the incident scene 110. At block 504,the electronic processor 205 retrieves (for example, from the memory 210or the database 106) the object of interest 112 based on the incidentidentifier. In some embodiments, the incident identifier is supplied bythe investigator. In some embodiments, the incident identifier isdetermined automatically based on the location of the portableelectronic device 102 as compared to the location of the incident scene110. In some embodiments, the incident identifier is determined andreceived from a computer aided dispatch system. In some embodiments,when an object of interest is nearby but not in the video stream, theelectronic processor 205 may display an indicator of which direction topoint the portable electronic device 102 in order to bring the object ofinterest 112 into view. In some embodiments, the second video stream 602is not captured. In such embodiments, the incident scene 110 may bereconstructed based on the location of the incident scene 110 (forexample, as provided by the incident scene identifier) and a directionin which the portable electronic device 102 is pointed.

At block 506, the electronic processor 205 superimposes or otherwisecombines, on the second video stream 602, based on the object location,the object of interest 112 to create a superimposed video stream 604 ofthe incident scene 110. The investigator (or any other user) viewing theincident scene 110 on the portable electronic device 102, can now seethe object of interest 112 as it appeared at the time of the capture ofthe first video stream 402. As illustrated in FIG. 6B, the portableelectronic device 102 presents an augmented reality view of the incidentscene 110. In some embodiments, the electronic processor 205 may usethree-dimensional information captured for the object of interest 112 todisplay a three-dimensional model of the object of interest 112. In someembodiments, if three-dimensional information is incomplete, the modelmay be completed using data retrieved based on the object class or themetadata 250. For example, if the object class and the metadata 250indicate that the object of interest 112 is a particular make, model,and year of vehicle, an automotive database might be queried to retrieveimages and dimensional information for that vehicle. In someembodiments, the object of interest 112 may be displayed on a 2D map ofthe incident scene 110.

As illustrated in FIG. 6B, the object of interest is displayed with anannotation indicator 606. The annotation indicator 606 indicates thatthe object of interest 112 has annotations associated with it. Theannotation indicator 606 may be an icon, for example, including atitle(s) and the type(s) of annotation(s) associated with the object ofinterest 112. At block 508, the electronic processor 205 receives asecond input selecting annotation indicator in the superimposed videostream 604. The second input may be, for example, a tap or touch on thedisplay 245. At block 510, the electronic processor 205 retrieves theannotation input 255, the metadata 250, or both, based on the secondinput. For example, the electronic processor 205 retrieves, from thedatabase 106, the annotation input 255 and the metadata 250 stored usingthe method 300, based on the object identifier for the object ofinterest 112. In some embodiments, the annotation input 255 may be showndirectly (for example, when the annotation consists of a short note).

At block 512, the electronic processor 205 displays the annotation input255 and the metadata 250 for the object of interest 112. In someembodiments, the metadata 250 includes a current location for the objectof interest 112. For example, the metadata may indicate that a weaponfrom a crime scene has been stored in a particular evidence locker atthe police station, or the metadata may indicate that a vehicle is inthe police impound lot.

In some instances, an investigator may return to re-investigate theincident scene 110 after the initial investigation is complete, butbefore the real-world object of interest 112 has been removed.Accordingly, FIG. 7 illustrates an example method 700 for highlighting avisual change in an image of an object of interest 112 located at anincident scene 110. The method 700 is described with respect to FIGS. 8Aand 8B, which illustrate the incident scene 110 and a graphical userinterface displayed on the portable electronic device 102. The method700 is described as being performed by the portable electronic device102 and, in particular, the electronic processor 205. However, it shouldbe understood that in some embodiments, portions of the method 700 maybe performed by other devices, including for example, the server 104.

At block 702, the electronic processor 205 and the camera 240 capture asecond video stream 802 of the incident scene 110. At block 704, theelectronic processor 205 displays the second video stream 802 on thedisplay 245. At block 706, the electronic processor 205 locates, in thesecond video stream 802, the object of interest 112, as describe abovewith respect to the method 300. At block 708, the electronic processor205 determines the object identifier based on the object of interest112, as described above.

At block 710, the electronic processor 205 retrieves (for example, fromthe memory 210, the database 106, or both) the annotation input 255 andthe metadata 250 based on the object identifier. At block 712, theelectronic processor 205 displays the annotation input 255 and themetadata 250 for the object of interest 112.

At block 712, the electronic processor 205 identifies a visual change804 in the object of interest 112. For example, the electronic processor205 may use image processing techniques to compare the object ofinterest 112 from the first video stream 402 with the object of interest112 in the second video stream 802 to determine if any portions of theobject of interest 112 have changed since the first video stream 402 wascaptured (for example, by comparing the time stamp to the current time).In some embodiments, the electronic processor 205 identifies a change instate (for example, the size, shape, location, color, the presence ofsmoke, a door or window is now open or closed, and the like) for theobject of interest 112. In some embodiments, the electronic processor205 identifies a change as something (for example, a license plate)missing from the object of interest.

At block 716, the electronic processor 205 highlights, on the object ofinterest 112, the change. For example, as illustrated in FIG. 8B, thechange may be shaded to highlight the area.

In some embodiments, the electronic processor 205 displays, on thedisplay 245, a timeline based on the incident scene 110 and the timestamp. For example, the timeline may display the time stamp on one end,the current time on the other end, and hash marks in between notingdivisions of time (for example, hours). In such embodiments, theelectronic processor may receive an input selecting a selected time (forexample, a tapping of one of the hash marks) on the timeline, and updatethe object of interest 112, the annotation input 255, and the metadata250 based on the selected time. For example, any updates since theselected time to those items may not be displayed, or may be greyed outto indicate that they are not applicable to the currently-selected time.

In some embodiments, the electronic processor 205 may display otherinformation related to the incident scene 110. For example, theelectronic processor 205 may display an incident scene perimeter, or thelocation(s) of other personnel at or relative to the incident scene 110.

In some cases, an investigator may want to indicate whether a line ofsight exists between two points in the incident scene 110. For example,in a crime scene, it may be advisable to know whether a line of sightexists between where a suspect was located and where a victim waswounded by gunfire. Accordingly, FIG. 9 illustrates an example method900 for annotating the incident scene 110. The method 900 is describedwith respect to FIG. 10 , which illustrates the incident scene 110 and agraphical user interface displayed on the portable electronic device102. The method 900 is described as being performed by the portableelectronic device 102 and, in particular, the electronic processor 205.However, it should be understood that in some embodiments, portions ofthe method 900 may be performed by other devices, including for example,the server 104.

At block 902, the electronic processor 205 receives an inputcorresponding to a first location 1002 at the incident scene 110. Atblock 904, the electronic processor 205 receives an input correspondingto a second location 1004 at the incident scene 110. The inputs receivedmay be, for example, taps or touches on the display 245.

A line of sight is an unobstructed path between the first location 1002and the second location 1004. At block 906, the electronic processor 205determines a line of sight 1006 based on the first location 1002 and thesecond location 1004. The line of sight 1006 is determined, for example,through image analysis and range imaging.

At block 908, the electronic processor determines a distance 1008between the first location 1002 and the second location 1004. In someembodiments, the distance 1008 is determined similarly to determiningthe relative location for the object of interest 112, as described abovewith respect to the method 300. At block 910, the electronic processor205 displays, on the display 245, the line of sight 1006 and thedistance 1008.

It should be noted that the systems and methods described above refer toa single incident scene 110 and a single object of interest 112.However, the systems and methods apply to multiple incident scenes andmultiple objects of interest. For example, selecting an object ofinterest, as described above, may apply to selecting one from severalobjects of interest displayed. It should also be noted that the systemsand methods presented herein are applicable outside of public safetyfield. For example, public or private utility workers may use thesystems and methods described above to identify and annotateinfrastructure objects (for example, utility poles, fire hydrants,transformers, control boxes, and the like). In another example,construction workers may use the systems and methods described above toidentify and annotate objects of interest at construction sites (forexample, by noting items needing attention for the next shift or crewcoming in).

In the foregoing specification, specific embodiments have beendescribed. However, one of ordinary skill in the art appreciates thatvarious modifications and changes can be made without departing from thescope of the invention as set forth in the claims below. Accordingly,the specification and figures are to be regarded in an illustrativerather than a restrictive sense, and all such modifications are intendedto be included within the scope of present teachings.

The benefits, advantages, solutions to problems, and any element(s) thatmay cause any benefit, advantage, or solution to occur or become morepronounced are not to be construed as a critical, required, or essentialfeatures or elements of any or all the claims. The invention is definedsolely by the appended claims including any amendments made during thependency of this application and all equivalents of those claims asissued.

Moreover in this document, relational terms such as first and second,top and bottom, and the like may be used solely to distinguish oneentity or action from another entity or action without necessarilyrequiring or implying any actual such relationship or order between suchentities or actions. The terms “comprises,” “comprising,” “has,”“having,” “includes,” “including,” “contains,” “containing” or any othervariation thereof, are intended to cover a non-exclusive inclusion, suchthat a process, method, article, or apparatus that comprises, has,includes, contains a list of elements does not include only thoseelements but may include other elements not expressly listed or inherentto such process, method, article, or apparatus. An element proceeded by“comprises . . . a,” “has . . . a,” “includes . . . a,” or “contains . .. a” does not, without more constraints, preclude the existence ofadditional identical elements in the process, method, article, orapparatus that comprises, has, includes, contains the element. The terms“a” and “an” are defined as one or more unless explicitly statedotherwise herein. The terms “substantially,” “essentially,”“approximately,” “about” or any other version thereof, are defined asbeing close to as understood by one of ordinary skill in the art, and inone non-limiting embodiment the term is defined to be within 10%, inanother embodiment within 5%, in another embodiment within 1% and inanother embodiment within 0.5%. The term “coupled” as used herein isdefined as connected, although not necessarily directly and notnecessarily mechanically. A device or structure that is “configured” ina certain way is configured in at least that way, but may also beconfigured in ways that are not listed.

It will be appreciated that some embodiments may be comprised of one ormore generic or specialized processors (or “processing devices”) such asmicroprocessors, digital signal processors, customized processors andfield programmable gate arrays (FPGAs) and unique stored programinstructions (including both software and firmware) that control the oneor more processors to implement, in conjunction with certainnon-processor circuits, some, most, or all of the functions of themethod and/or apparatus described herein. Alternatively, some or allfunctions could be implemented by a state machine that has no storedprogram instructions, or in one or more application specific integratedcircuits (ASICs), in which each function or some combinations of certainof the functions are implemented as custom logic. Of course, acombination of the two approaches could be used.

Moreover, an embodiment can be implemented as a computer-readablestorage medium having computer readable code stored thereon forprogramming a computer (e.g., comprising a processor) to perform amethod as described and claimed herein. Examples of suchcomputer-readable storage mediums include, but are not limited to, ahard disk, a CD-ROM, an optical storage device, a magnetic storagedevice, a ROM (Read Only Memory), a PROM (Programmable Read OnlyMemory), an EPROM (Erasable Programmable Read Only Memory), an EEPROM(Electrically Erasable Programmable Read Only Memory) and a Flashmemory. Further, it is expected that one of ordinary skill,notwithstanding possibly significant effort and many design choicesmotivated by, for example, available time, current technology, andeconomic considerations, when guided by the concepts and principlesdisclosed herein will be readily capable of generating such softwareinstructions and programs and ICs with minimal experimentation.

The Abstract of the Disclosure is provided to allow the reader toquickly ascertain the nature of the technical disclosure. It issubmitted with the understanding that it will not be used to interpretor limit the scope or meaning of the claims. In addition, in theforegoing Detailed Description, it can be seen that various features aregrouped together in various embodiments for the purpose of streamliningthe disclosure. This method of disclosure is not to be interpreted asreflecting an intention that the claimed embodiments require morefeatures than are expressly recited in each claim. Rather, as thefollowing claims reflect, inventive subject matter lies in less than allfeatures of a single disclosed embodiment. Thus the following claims arehereby incorporated into the Detailed Description, with each claimstanding on its own as a separately claimed subject matter.

We claim:
 1. A system for displaying an image of an object of interestlocated at an incident scene, the system comprising: an image capturedevice; a display; a memory; and an electronic processor coupled to theimage capture device, the display, and the memory, and configured toreceive, from the image capture device, a first video stream of theincident scene captured at a first time; display the first video streamon the display; receive an input indicating a pixel location in thefirst video stream; detect the object of interest in the first videostream based on the pixel location; determine an object class for theobject of interest; determine an object identifier for the object ofinterest; determine metadata for the object of interest including theobject class, an object location relative to the image capture devicewithin the incident scene, a unique incident identifier from acomputer-aided dispatch system and corresponding to the incident scene,and a time stamp; receive an annotation input for the object ofinterest; associate the annotation input and the metadata with theobject identifier; store, in the memory, the object of interest, theannotation input, and the metadata; receive, from the image capturedevice, a second video stream of the incident scene at a second time,the second time being different from the first time; display the secondvideo stream on the display; locate, in the second video stream, theobject of interest; determine the object identifier based on the objectof interest; retrieve, from the memory, the annotation input and themetadata based on the object identifier; display the annotation inputand the metadata for the object of interest; identify a visual change ina portion of the object of interest; and highlight, on the object ofinterest, the visual change.
 2. The system of claim 1, wherein theelectronic processor is further configured to detect the object ofinterest in the first video stream using an object classifier; anddetermine the object class using the object classifier.
 3. The system ofclaim 1, wherein the electronic processor is further configured toretrieve, from the memory, the object of interest based on the incidentidentifier; superimpose, on the second video stream, based on the objectlocation, the object of interest to create a superimposed video streamof the incident scene; receive a second input selecting an annotationindicator in the superimposed video stream; retrieve the annotationinput and the metadata based on the second input; and display theannotation input and the metadata for the object of interest.
 4. Thesystem of claim 3, wherein the metadata includes a current location forthe object of interest.
 5. The system of claim 1, wherein the electronicprocessor is further configured to display, on the display, a timelinebased on the incident scene and the time stamp; receive a second inputselecting a selected time on the timeline; and update the object ofinterest, the annotation input, and the metadata based on the selectedtime.
 6. The system of claim 1, wherein the electronic processor isfurther configured to receive a second input selecting the object ofinterest; and in response to the second input, display an executablemenu based on at least one selected from a group consisting of theincident identifier, the object identifier, and the metadata; whereinthe executable menu performs one of entering an edit mode and displayingdetailed annotations.
 7. The system of claim 1, wherein the electronicprocessor is further configured to receive an input corresponding to afirst location at the incident scene; receive an input corresponding toa second location at the incident scene; determine a line of sight basedon the first location and the second location; determine a distancebetween the first location and the second location; and display, on thedisplay, the line of sight and the distance.
 8. The system of claim 1,wherein the display is a head-mounted display.
 9. A method fordisplaying an image of an object of interest located at an incidentscene on a display of a device, the method comprising: receiving, froman image capture device, a first video stream of the incident scenecaptured at a first time; displaying the first video stream on thedisplay; receiving an input indicating a pixel location in the firstvideo stream; detecting the object of interest in the first video streambased on the pixel location; determining an object class for the objectof interest; determining an object identifier for the object ofinterest; determining metadata for the object of interest including theobject class, an object location relative to the image capture devicewithin the incident scene, a unique incident identifier from acomputer-aided dispatch system and corresponding to the incident scene,and a time stamp; receiving an annotation input for the object ofinterest; associating the annotation input and the metadata with theobject identifier; storing, in a memory, the object of interest, theannotation input, and the metadata; receiving, from the image capturedevice, a second video stream of the incident scene at a second time,the second time being different from the first time; displaying thesecond video stream on the display; locating, in the second videostream, the object of interest; determining the object identifier basedon the object of interest; retrieving, from the memory, the annotationinput and the metadata based on the object identifier; displaying theannotation input and the metadata for the object of interest;identifying a visual change in the object of interest; and highlighting,on the object of interest, the visual change.
 10. The method of claim 9,wherein detecting the object of interest includes detecting the objectof interest using an object classifier; and determining an object classfor the object of interest includes determining an object class usingthe object classifier.
 11. The method of claim 9, wherein receiving anannotation input for the object of interest includes receiving at leastone of a text-based annotation and an audio annotation.
 12. The methodof claim 9, further comprising: superimposing, on the second videostream, based on the object location, the object of interest to create asuperimposed video stream of the incident scene; receiving a secondinput selecting an annotation indicator in the superimposed videostream; retrieving the annotation input and the metadata based on thesecond input; and displaying the annotation input and the metadata forthe object of interest.
 13. The method of claim 12, wherein displayingthe metadata for the object of interest includes displaying a currentlocation for the object of interest.
 14. The method of claim 9, furthercomprising: displaying, on the display, a timeline based on the incidentscene and the time stamp; receiving a second input selecting a selectedtime on the timeline; and updating the object of interest, theannotation input, and the metadata based on the selected time.
 15. Themethod of claim 9, further comprising: receiving a second inputselecting the object of interest; and in response to the second input,displaying an executable menu based on at least one selected from agroup consisting of the incident identifier, the object identifier, andthe metadata; wherein the executable menu performs one of entering anedit mode and displaying detailed annotations.
 16. The method of claim9, further comprising: receiving an input corresponding to a firstlocation at the incident scene; receiving an input corresponding to asecond location at the incident scene; determining a line of sight basedon the first location and the second location; determining a distancebetween the first location and the second location; and displaying, onthe display, the line of sight and the distance.